Surgeon head-mounted display apparatuses

ABSTRACT

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

RELATED APPLICATIONS

The present patent application is continuation of U.S. patentapplication Ser. No. 16/706,948 which is a continuation of U.S. patentapplication Ser. No. 15/699,273 which is a continuation of U.S. patentapplication Ser. No. 15/013,594 filed on Feb. 2, 2016, which claims thebenefit of priority from U.S. Provisional Patent Application No.62/111,379, filed on Feb. 3, 2015, from U.S. Provisional PatentApplication No. 62/181,433, filed on Jun. 18, 2015, and from U.S.Provisional Patent Application No. 62/270,895, filed on Dec. 22, 2015,the disclosure and content of all of which are incorporated by referenceherein in their entireties.

TECHNICAL FIELD

The present disclosure relates to surgical operating room equipment andprocedures and, more particular, to generating anatomical models andother information to be displayed as augmenting reality in surgicaloperating rooms.

BACKGROUND

It has become commonplace in operating rooms (ORs) for surgeons andother personnel to refer to many different types of visual patient dataduring surgical procedures. For example, a surgeon may refer to digitalphotographs and video from magnetic resonance imaging equipment,computed tomography scanning equipment, x-ray equipment,three-dimensional ultrasound equipment, endoscopic equipment, 3Dcomputer modeling equipment, patient monitoring equipment, medicalrecords databases, and other equipment during a surgical procedure.

ORs therefore typically include many display devices positioned atvarious locations that are expected to be viewable during a procedure.Personnel may refer to display devices hung from a ceiling, mounted froma wall, supported on a cart, etc. However, it is difficult or notpossible to position the display devices for convenient viewing by allnecessary personnel. Mentally translating between the orientation ofinformation shown on display devices that are angularly offset from theorientation of a patient's body can be particularly difficult, prone toerrors, or inefficiently time-consuming for personnel during aprocedure. Moreover, frequent shifting of focal reference from a patientsurgical site to remotely located display devices may result in fatigueand have a deleterious effect on quality of the procedure.

Another difficulty surgeons and other personnel have is relating what isviewed on the display devices to precise locations on a patient. Forexample, it can be difficult for a surgeon to identify where aparticular location within a displayed x-ray or other image correspondsto on the patient. Moreover, a surgeon may use a 3D anatomical model topractice before a procedure, but may not be able to effectively use themodel during surgery because of the inherent difficulty of relating themodel to the patient in real-time.

SUMMARY

Some embodiments of the present disclosure are directed to an augmentedreality surgical system that includes a head mounted display, a motionsensor, at least one camera, and computer equipment. The head mounteddisplay includes a see-through display screen that display images whileallowing transmission of ambient light therethrough. The motion sensoris connected to the head mounted display and configured to output a headmotion signal indicating measured movement of the head mounted display.The at least one camera is configured to observe reference markersconnected to the head mounted display, reference markers connected to apatient, and reference markers connected to a surgical tool locatedwithin a surgical room. The computer equipment is configured to computethe relative location and orientation of the reference markers connectedto the head mounted display and the reference markers connected to thepatient based on processing a video signal from the at least one camera.The computer equipment is further configured to generate a threedimensional anatomical model using patient data created by medicalimaging equipment that has imaged a portion of the patient, and torotate and scale at least a portion of the three dimensional anatomicalmodel based on the relative location and orientation of the referencemarkers connected to the head mounted display and the reference markersconnected to the patient, and further rotate the at least a portion ofthe three dimensional anatomical model based on the head motion signalto track measured movement of the head mounted display. The computerequipment is further configured to generate a video signal based on therotated and scaled three dimensional anatomical model, and to output thevideo signal to the display screen of the head mounted display.

Some other embodiments of the present disclosure are directed to anaugmented reality surgical system for displaying multiple video streamsto a user. The augmented reality surgical system includes a head mounteddisplay, a motion sensor, and computer equipment. The head mounteddisplay includes a see-through display screen that display images whileallowing transmission of ambient light therethrough. The motion sensoris connected to the head mounted display and configured to output a headmotion signal indicating measured movement of the head mounted display.The computer equipment is configured to receive a plurality of videostreams from one or more source devices and to control which of thevideo streams are output as a video signal to the display screen basedon the head motion signal.

Other systems, methods, and computer program products according toembodiments of the inventive subject matter will be or become apparentto one with skill in the art upon review of the following drawings anddetailed description. It is intended that all such additional systems,methods, and computer program products be included within thisdescription, be within the scope of the present inventive subjectmatter, and be protected by the accompanying claims. Moreover, it isintended that all embodiments disclosed herein can be implementedseparately or combined in any way and/or combination.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features of embodiments will be more readily understood from thefollowing detailed description of specific embodiments thereof when readin conjunction with the accompanying drawings, in which:

FIGS. 1-8 illustrate a head mounted display apparatus that can be wornon a user's head and operates according to some embodiments of thepresent disclosure;

FIG. 9 is a block diagram of electronic components of a computer systemthat includes a head mounted display apparatus configured according tosome embodiments of the present disclosure;

FIGS. 10-12 illustrate operations and methods that may be performed by asystem that includes a head mounted display apparatus to control thedisplay of virtual display panels through a display screen of a headmounted display apparatus according to some embodiments of the presentdisclosure;

FIG. 13 is a block diagram of components of an augmented realitysurgical system that tracks the location of surgical tools, a surgeon'shead mounted display, and parts of a patient's anatomy, and generates athree dimensional (3D) model from patient data that is displayed on thehead mounted display to be rendered super-imposed at a visually alignedlocation on the patient's body in accordance with some embodiments ofthe present disclosure;

FIG. 14 is another block diagram of the electronic components of theaugmented reality surgical system of FIG. 13 according to someembodiments of the present disclosure;

FIG. 15 is another block diagram that illustrates further exampleoperations of electronic system subcomponents of the augmented realitysurgical system of FIG. 13 according to some embodiments of the presentdisclosure;

FIGS. 16 and 17 are block diagrams that illustrate further exampleoperations of electronic system subcomponents that can be included inthe augmented reality surgical system of FIG. 13 according to some otherembodiments of the present disclosure;

FIG. 18 illustrates a graphical image generated on the head mounteddisplay showing a virtual trajectory of a surgical tool into a patient'sanatomy relative to an anatomical model generated from 3D computerizedtomography (CT) scan data in accordance with some embodiments of thepresent disclosure;

FIG. 19 illustrates another graphical image generated on the headmounted display showing a cross-sectional slice along plane 19-19 inFIG. 18 rotated to provide a front view;

FIG. 20 illustrates another graphical display generated on the headmounted display showing a sequence of cross-sectional slices of the 3DCT scan anatomical model spaced apart along the virtual trajectory ofthe tool and illustrates points of intersection between the virtualtrajectory and the slices;

FIG. 21 illustrates another graphical display generated on the headmounted display showing a sequence of cross-sectional slices of the 3DCT scan anatomical model spaced apart along the virtual trajectory ofthe tool and oriented in planes perpendicular to the virtual trajectoryof the tool; and

FIGS. 22 and 23 each illustrate a graphical image generated on the headmounted display showing the cross-sectional slices along planes 22-22and 23-23, respectively, in FIG. 21 rotated to provide a front view.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of embodiments of thepresent disclosure. However, it will be understood by those skilled inthe art that the present invention may be practiced without thesespecific details. In other instances, well-known methods, procedures,components and circuits have not been described in detail so as not toobscure the present invention. It is intended that all embodimentsdisclosed herein can be implemented separately or combined in any wayand/or combination.

Embodiments of the present disclosure are directed to an augmentedreality surgical system that includes a head mounted display (HMD)apparatus that can be worn by a surgeon, physician, or other personnelduring a medical procedure. The HMD can be configured to providelocalized, real-time situational awareness to the wearer. The HMDincludes a display screen that can be positioned within theline-of-sight and/or periphery Field Of View (FOV) of the wearer toprovide visual information that can be organized and displayed as asingle virtual display or as a collection of virtual displays that awearer can navigate between to view using head movement, hand gestures,voice commands, eye control, and/or other operations disclosed herein.

A surgeon or other person can wear the HMD to see a graphicalrepresentation of what is within the patient's body but covered fromview by skin, muscle, organs, skeletal structure, etc. Using the HMD canenable a surgeon to minimize the size of an incision by observing wherethe incision needs to be made to reveal a targeted portion of the body.Similarly, the HMD can be used when replacing a bone with prosthesis toenable a surgeon to observe an exact positioning reference that aidswith orienting and moving surgical tools and the prosthesis during theprocedure. The HMD may operate to improve the efficiency, productivity,throughput, and/or accuracy, and/or safety of the wearer's performanceof a medical procedure. Moreover, the HMD can reduce mental fatigue byreducing or eliminating a need for the wearer to reference remotedisplay devices having substantial angular offsets during a medicalprocedure.

Although various embodiments of systems having HMDs are described foruse in the environment of a surgical operating room, they may be used inother applications. For example, the systems may be used withinindustrial environments such as warehousing applications, manufacturingapplications, product inspection applications, and/or maintenanceapplications.

Example Head Mounted Display Apparatuses

FIG. 1 illustrates a HMD apparatus 100 (also “HMD 100” for brevity)configured according to some embodiments of the present disclosure.Referring to FIG. 1, the HMD 100 includes a semitransparent displayscreen 110 connected to a display module that processes and displaysvideo and other images on the display screen 110 (e.g., a LCD display, areflective screen on which the display module projects images, etc.) forviewing by a user. The display module may be within a housing 118 of theHMD 100 or may be contained within a communicatively connected computersystem.

In the illustrated embodiment, the HMD 100 is mounted to a headband 120and positioned so that the display screen 110 extends within theperipheral vision of the user. The housing 118 encloses electroniccomponents that display information on the display screen 110 and mayoperate in combination with a remote but communicatively connectedcomputer equipment and/or with computer equipment integrated within thehousing 118 to sense and interpret movement of the head, sense andinterpret gestures made by a user's hands or other objects, and/or senseand interpret voice commands by the user. The display screen 110 canprovide a monocular see-through display or a stereo set of see-throughdisplays so that the user can view information displayed on the displaywhile looking through the display to view other objects. The headband120 may include a headlamp, camera, or other apparatus that can be wornon a user's head.

The user is illustrated as wearing glasses 150 that includethrough-the-lens (TTL) loupes 152, protruding from lenses of the glasses150, that provide magnified viewing to the user. The display screen 110extends downward from the housing 118 and is positionable by the user tobe in the user's field-of-view or immediately adjacent to the TTL loupes152 within the user's peripheral vision. The display screen 110 can be asee-through display device allowing the user to see video superimposedon the environment seen through the display screen 110.

The TTL loupes 152 may not be included in the HMD 100 when the displayscreen 110 is configured to be in the direct line-of-sight of the user.Alternatively, the display screen 110 can be positioned adjacent to theTTL loupes 152 so that the user can make a minor upward shift in eyeline-of-sight from looking through the TTL loupes 152 to instead viewinformation displayed on the display screen 110. In some embodiments thedisplay screen 110 may be incorporated within one or both TTL loupes 152so that the user can look through the TTL loupes 152 to view graphicalimages super-imposed on Objects within the FOV of the TTL loupe 152. TheHMO 100 may be configured to be attachable to any type of eyeglassframes, including prescription glasses, protective glasses, frameswithout lenses, transparent or protective shields, etc.

The display screen 110 can be moved by a user to a location providingconvenient visual reference through a two-arm friction joint linkage 112that provides telescopic and up-and-down adjustment of location of thehousing 118. A ball-and-socket joint 114 is connected between thelinkage 112 and the housing 118 to provide planar adjustment for thedisplay screen 110. A pivot joint 116 connected between theball-and-socket joint 114 and the housing 118 allows the user to pivotthe housing 116 and connected display screen 110. The display screen 110can thereby be flipped-up outside the user's peripheral vision when notbeing used.

The HMD 100 may include sensors such as inertial sensors or othersensors, such as a gyroscope, accelerometer (e.g., a multi-axisaccelerometer), and/or magnetometer that output a signal indicating ameasurement of movement or static orientation of the user's head whilewearing the HMD 100. For example, the motion sensor may output a headmotion signal that indicates yaw (i.e., rotation of the user's head leftor right), pitch (i.e., rotation of the user's head up or down), and/orroll (i.e., side-to-side tilt of the user's head). The sensors may bespaced apart on the headband 120 or enclosed within the housing 118.

The HMD 100 may include at least one camera facing away from the userthat outputs video and/or other images for processing and relay to otherHMDs 100 worn by other personnel assisting with the procedure, to otherdisplay devices, and/or to a video server for storage. For example, thecamera may be configured to be aligned with the user's line-of-sightwhen the user has adjusted the display screen 110 to be comfortablyviewed by the user. When more than one camera is connected to the HMD100, video streams from the cameras can be provided to an operationalfunction that estimates distance to an object viewed by the cameras. Theoperational function can include triangulation of distance to the objectbased on angular offset of the object viewed in the video streams and aknown distance between the cameras.

The at least one camera may be connected to a gesture interpretationmodule configured to sense gestures made by a user's hands or otherobjects, recognize a gesture as corresponding to one of a plurality ofdefined commands, and trigger operation of the command. The HMD 100 mayinclude a microphone connected to a voice interpretation moduleconfigured to recognize a received voice command as corresponding to oneof a plurality of defined voice commands, and trigger operation of thecommand.

The headband 120 may have a plurality of attachment points whereinertial sensors, camera, microphone, etc. can be releasably attached.Some of the attachment points may have rigid supporting structuresbetween them to maintain a defined physical alignment between theattached inertial sensors, etc.

FIG. 2 illustrates a side view of another HMD 200 with a display screen210 and electronic components 214 (shown without a housing) which areconfigured according to some embodiments of the present disclosure. Thedisplay screen 210 extends downward from the electronic components 214to be in the user's line-of-sight or immediately adjacent TTL loupes 152within the user's peripheral vision. The electronic components 214 areconnected to the headband 120 via a pivot 212 allowing the electroniccomponents 214 and connected display screen 210 to be flipped-down to adeployed position as shown in FIG. 2 and flipped-up to a stored positionwhen the user does not desire to view the display screen 210.

FIG. 3 illustrates another HMD 300 configured according to someembodiments of the present disclosure. The HMD 300 includes a displayscreen illustrated behind a protective shield 310 that extends downwardfrom a housing 318 enclosing electronic components. The display screenand/or the protective shield 310 may include a coating that providesvariable contrast to enhance viewability of displayed information whilesubject to a range of ambient brightness. The protective shield 310 mayprovide a variable focal point (diopter). The protective shield 310 canhe flipped from a stored up-position to a protective down-position (asshown in FIG. 3) to cover an outside surface of the display screen thatfaces a patient and function to protect the display screen from fluidsand other materials occurring during a procedure. The display screen canbe moved by a user through a two-arm friction joint linkage 312 thatprovides telescopic and up-and-down adjustment of location of thehousing 318 to enable a user to position the display screen at alocation providing convenient visual reference. A ball-and-socket joint316 is connected between the linkage 312 and the housing 118 to provideplanar adjustment for the display screen. The linkage 312 is connectedto the headband 120 through a pivot joint 314 to allow the user to flipthe housing 318 and connected display screen up and down. The displayscreen can thereby be flipped-up outside the user's line-of-sight or theuser's peripheral vision when not being used.

FIG. 4 illustrates another HMD 400 configured according to someembodiments of the present disclosure. The HMD 400 includes a displayscreen 410 that extends downward from a housing 418 enclosing electroniccomponents. The display screen 410 and housing 418 are connected to aball-and-socket joint 416 which provides planar adjustment for thedisplay screen 410. The ball-and-socket joint 416 is connected to apivot 414 that allows the housing 418 and connected display screen 410to be pivoted up and down, so that the display screen 410 can beflipped-up outside the user's line-of-sight or the user's peripheralvision when not being used. The pivot 414 is connected to a sliding arm412 that connects to the headband 120. The sliding arm 412 providestelescoping adjustment to allow user placement of the display screen 410a desired distance from an eye.

FIG. 5 illustrates a front view of the HMD 400 of FIG. 4 with thehousing 418 removed to expose printed circuit boards (PCBs) 450 whichoperationally connect the electronic components mounted thereon. Theelectronic components display images on the display screen 410, and mayoperate in combination with integrated or remote computer equipment tosense and interpret movement of the head, sense and interpret gesturesmade by a uses hands. eyes, or other objects, and/or sense and interpretvoice commands by the user. The PCBs 450 are tilted at a definednon-zero angle relative to vertical to reduce the profile cross-sectionof the housing 418. For example, the PCBs 450 can extend generallydiagonally across the housing 418.

FIG. 6 illustrates another HMD 500 having a single display screenconnectable to an eyeglass frame to provide monocular viewing by theuser. FIG. 7 illustrates another HMD 502 including a pair of displayscreens that are connectable to opposite sides of an eyeglass frame toprovide binocular viewing. Although the display screens in FIGS. 6 and 7are shown as being opaque, they may instead allow a user to see throughthe display screen while viewing information displayed thereon.

FIG. 8 illustrates example design parameters that may be used whenconfiguring a HMD to allow movement of the display screen to accommodateanticipated variation in user head geometries. Referring to FIG. 8, thespread between the minimum and maximum eye position inward from theforehead is about 10 mm. Consequently, the HMD may preferably beconfigured to allow variation in the distance between the display screenand a user's eye of up to 27 mm and, more preferably, 13 mm.

Example Computer System Incorporating a Head Mounted Display Apparatus

FIG. 9 is a block diagram of electronic components of a computer systemthat includes a HMD apparatus 600, computer equipment 620, and asurgical video server 650. The video server 650 can be connected via adata network 640 to a patient database 642, imaging equipment 644, andother electronic equipment 646. The HMD 600 may correspond to any of theHMDs of FIGS. 1-8. Although the computer equipment 620 is illustrated asbeing separate from the HMD 600, some or all of the operations disclosedherein as being performed by the computer equipment 620 may additionallyor alternatively be performed by one or more processors residing withinthe HMD 600. Similarly, some of the operations disclosed herein as beingperformed by the HMD 600 may additionally or alternatively be performedby one or more processors residing within the computer equipment 620.

The video server 650 can receive, store, and route information, videostreams between the patient database 642, imaging equipment 644, andother electronic equipment 646 and the HMD 600. As used herein, a videostream can include any type of information that can be provided to adisplay device for display, including without limitation a still image(e.g., digital photo), a sequence of still images, and video havingframes provided at a defined frame rate. The imaging equipment 644 mayinclude endoscope cameras, magnetic resonance imaging equipment,computed tomography scanning equipment, three-dimensional ultrasoundequipment, endoscopic equipment, and/or computer modeling equipmentwhich can generate multidimensional (e.g., 3D) model based on combiningimages from imaging equipment. The patient database 642 can retrievablystore information relating to a patient's medical history, and may storepatient images from earlier procedures conducted via the imagingequipment 644. The other equipment 646 may provide information relatingto real-time monitoring of a patient, including, for example,hemodynamic, respiratory, and electrophysiological signals.

The computer equipment 620 operationally interfaces the HMD 600 to thevideo server 650. The computer equipment 620 includes a video capturecard 622 that can simultaneously receive a plurality (N) of videostreams and information (e.g., textual descriptions, audio signals,etc.) from the video server 650 and/or directly from the imagingequipment 644, the patient database 642, and/or the other equipment 646.The computer equipment 620 may communicate with the video server 650,the HMD 600, and other equipment of the system via a wireless and/orwired network interface 628 using any appropriate communication medium,including but not limited to a wireless air interface (e.g., 3GPP LongTerm Evolution (LTE), WLAN (IEEE 802.11), WiMax, etc.), wireline,optical fiber cable, or any combination thereof. In the exampleembodiment of FIG. 9 the video capture card 622 simultaneously receivesup to 4 video streams via 4 interfaces. in one embodiment the HMD 600 iscommunicatively connected to the computer equipment 620 via an HDMIcable, a USB or RS 422 cable connected to the motion sensor 604 and/orgesture sensor 602, and a USB 3.0 or firewire cable connected to thecamera 610. A microphone 612 can be connected to the computer equipment620. The video and/or sensor signaling may alternatively be communicatedbetween the HMD 600 and the computer equipment 620 through a wirelessair interface, such as the network interface 628.

The HMD 600 includes a display module 606 that processes and displaysvideo and other images on the display screen 608 (e.g., a LCD display, areflective screen on which the display module 606 projects images, etc.)for viewing by a user. It may be preferable for the display screen 608to be a see-through display device allowing the user to see displayedvideo superimposed on what is viewed through the display screen 608. Thevideo streams received by the video capture card 622 are processed by agraphics processing unit (GPU) 638, conditioned by a display driver 614,and provided to the display module 606 for display on the display screen608. A symbol generator 624 may add graphical indicia and/or textualinformation to the video stream(s) provided to the HMD 600 based oninformation received from the video server 650 (e.g., via the patientdatabase 642).

The display driver 614 may reside in the computer equipment 620 or theHMD 600. In one embodiment, the display driver 614 receives video via aHDMI interface from the GPU 638, and converts the digital video signalto an analog video signal which is output as low-voltage differentialsignaling (LVDS) to the display module 606. The display driver 614 mayalso provide power and/or other signaling to the display module 606 viaa LED drive signal.

The HMD 600 can include a camera 610, or a plurality of the cameras 610,facing away from the user that outputs video and/or other images via thewireless and/or wired network interface 628, illustrated as a HDMI cablein FIG. 9, to the GPU 638 for processing and relay to the video server650 for storage and possible further relay to other HMDs 600 worn byother personnel assisting with the procedure. For example, the camera610 may be configured to be aligned with the user's line-of-sight whenthe user has adjusted the display screen 608 to be comfortably viewed bythe user. A video signal from the camera 610 can be processed throughthe computer equipment 620 and provided to the video server 650 forrecording what the user is viewing during the procedure and/or can beprovided as a real-time video stream to other HMDs 600 worn by personnelassisting with the procedure so that the personnel can observe what theuser is seeing. The video signal from the camera 610 may be augmented bythe symbol generator 624 with one or more designation symbols. Theaugmented symbols may, for example, identify the user as the source ofthe video stream and/or be added to a video stream by the user toidentify observed features, such as a patient's anatomy.

The HMD 600 may include a motion sensor 604 and/or a gesture sensor 602.The motion sensor 604 may be a gyroscope, accelerometer (e.g., amulti-axis accelerometer), and/or tilt sensor that outputs a head motionsignal indicating a measurement of movement of the user's head whilewearing the HMD 600. The motion sensor 604 may be powered by thecomputer equipment 620 and may output the head motion signal via acommunication interface, such as a RS-422 serial digital interface. Forexample, the motion sensor 604 may output a head motion signal thatindicates yaw movement (i.e., rotation of the user's head left or right)and/or indicates pitch movement (i.e., rotation of the user's head up ordown).

The motion sensor 604 may be a sourceless orientation sensor. The headmotion signal may be processed by the HMD 600 and/or by the computerequipment 620 to compensate for drift error introduced by the motionsensor 604. In one embodiment, one directional reference (e.g., yaw)component of the head motion signal is corrected toward zero responsiveto another reference component (e.g., pitch) of the head motion signalbeing within a threshold offset of a defined value. For example, yawdrift error in the head motion signal can be determined based onmonitoring yaw values of the motion signal while the user is lookingdown at a defined pitch (e.g., pitch being within a threshold range of adefined value) to align the user's eyes with an object (e.g., when asurgeon repetitively looks down to view a surgical site of a patient).In one embodiment, responsive to the pitch component of the head motionsignal indicating that a surgeon is looking down for at least athreshold time that is indicative of the surgeon visually concentratingon a surgical site, the computer equipment 620 assumes that the HMD 600is stabilized along the yaw axis and computes yaw drift error based onmeasured change in the yaw component over a defined time interval. Thehead motion signal is then compensated to remove the determined yawdrift error. In another embodiment, the computer equipment 620 measuresdrift in the yaw component of the head motion signal while a staticimage is displayed on the display screen, assuming that the surgeon'shead is stabilized along the yaw axis, and then compensates the headmotion signal to remove the measured drift in the yaw component.

The head motion signal may be processed by the HMD 600 and/or by thecomputer equipment 620 to identify an origin for one or more directionalreference components from which movement is referenced. For example, anorigin location from which yaw is measured may be identified based on anaverage (e.g., median or mode) of a yaw component of the head motionsignal during times when the user is looking down at a defined pitch toalign the user's eyes with an object (e.g., surgeon looking down to viewa surgical site).

The directional reference (e.g., pitch or yaw) of the head motionsignal, which is defined to trigger compensation for drift error and/orwhich is defined as a reference origin for movement measurement, may beidentified based on the user maintaining a substantially constantorientation of the HMD 600 for a threshold time (e.g., dwell time). Forexample, when a surgeon has maintained a relatively constant headposition while viewing a surgical site of a patient for a thresholdtime, the directional reference (e.g., pitch or yaw) of the head motionsignal during that dwell time can be used as a basis for compensatingfor drift error and/or setting as a reference origin for display ofvirtual display panels illustrated in FIGS. 10-12. In one embodiment,the head motion signal may be processed by the HMD 600 and/or by thecomputer equipment 620 to estimate gyroscope bias(es) giving rise to yawdrift and/or pitch drift accumulating over time based onpseudo-measurements of the yaw and/or the pitch provided by the headmotion signal Which is expected to be nearly zero each time the surgeonlooks down at the same surgical site and steadies the head to center theline-of-sight at a same location on the patient.

The gesture sensor 602 may include any type of sensor that can sense agesture made by a user. In a surgical environment, use of a gesturesensor 602 to receive a gesture-based command from a surgeon or other ORpersonnel can be advantageous because it avoids a need for the user totouch a non-sterile surface of the HMD 600 or other device. The gesturesensor 602 may include the camera 610 which outputs video (e.g., RGB-Dvideo) displaying movement of a user's hand, fingers, arms or otherobjects moved by the user along a pathway that the user knows willdefine a command identifiable by an operational surgical program (OSP)632 and/or another component of the system. The camera 610 or anothercamera. may be directed toward one of the user's eyes to identify adwell time of the eye, blink timing, and/or movement of the eye togenerate a command from the user to control what is displayed on thedisplay screen 608.

The gesture sensor 602 may alternatively or additionally include one ormore photoelectric motion and/or proximity sensors. In one embodiment,the gesture sensor 602 has a plurality of infrared emitters and aplurality of photodiodes. Adjacent pairs of an infrared emitter and aphotodiode are spaced apart and arranged to form a directional arrayfacing outward from a housing of the HMO 600 to sense presence of auser's hand adjacent to the army and/or to sense a direction of movementas the user's hand is moved across the array. A user may, for example,swipe a hand in a first direction across the array (without touching thehousing) to input a first type of gesture recognized by the OSP 632processed by the processor 626 which triggers a first type of operationby the OSP 632, swipe the hand in a second direction about opposite tothe first direction across the array to input a second type of gesturerecognized by the OSP 632 which triggers a second type of operation bythe OSP 632, swipe the hand in a third direction about perpendicular tothe first direction across the array to input a third type of gesturerecognized by the OSP 632 which triggers a third type of operation bythe OSP 632, and so on with other directions of movement beingidentifiable as other types of gestures provided by the user to triggerother types of operations by the OSP 632.

In another embodiment the gesture sensor 602 includes an ultrasonic echoranging transducer that senses signal echo reflections from a user'shand and outputs a signal to the processor 626 which identifies gesturesformed by movement of the hand. In another embodiment the gesture sensor602 includes a capacitive sensor that senses presence of a user's handthrough capacitive coupling between a charge plate and the user's hand.A plurality of capacitive sensors may be spaced apart to form thegesture sensor 602 and configured to sense a direction of movement ofthe uses hand relative to the array of charge plates (e.g., sense anorder with which plates experienced increased coupling to the user'shand). Different sensed directions of movement can be interpreted by theOSP 632 and/or another component of the system as representing differentcommands selected by the user for operation.

The HMD 600 can include a microphone 612 configured to receive voicecommands from a user. The processor 626 executing the OSP 632 and/oranother component of the system can be configured to recognize areceived voice command as corresponding to one of a plurality of definedvoice commands, and trigger operation of a command corresponding to therecognized voice command to control information displayed on the displayscreen 608.

The computer equipment 620 includes a general processor 626 and memory630. The processor 626 may include one or more data processing circuits,such as a general purpose processor and/or special purpose processor,such as a microprocessor and/or digital signal processor. The processor626 is configured to execute computer program code in the memory 630,described below as a non-transitory computer readable medium, to performat least some of the operations described herein. The computer programcode may include the OSP 632. The OSP 632 when executed by the processor626 causes the processor 626 to perform operations in accordance withone or more embodiments disclosed herein. The computer equipment 620 mayfurther include a speaker, a user input interface (e.g., touch screen,keyboard, keypad, etc.), a display device, etc.

In one embodiment, the video signal from the camera 610 is displayed onthe display device 608 of the same HMD 600, and the symbol generator 624in combination with the OSP 632 processed by the processor 626 mayoperate to display graphical indicia (e.g., reticle) that can bepositioned within a plane of the video stream by the user responsive torecognition of voice commands via the microphone 612, to track movementof the user's finger, hand, or other object recognized in the videostream responsive to the gesture sensor 602 (e.g., via the camera 610),and/or to track motion sensed by the motion sensor 604. The user maytrigger the OSP 632 to capture a still image from the video stream withthe incorporated graphical indicia responsive to a voice command, agesture sensed by the gesture sensor 602, and/or a motion sensed by themotion sensor 604. In this manner, a surgeon may view video of asurgical site and steer a graphical indicia to be aligned within thevideo overlapping a point-of-interest in the patient's anatomy, andtrigger capture of a still image including the video and graphicalindicia that can be saved on the video server 650 and/or distributed toanother HMD 600 for viewing by another surgeon or person.

In a further embodiment a user can view video from the camera 610 andsteer a graphical indicia to be aligned with a location within a planeof the video stream, and can trigger recordation of the presentpositional alignment of the camera 610. The OSP 632 may then operate tomaintain alignment of the graphical indicia displayed in the displayscreen 608 between, for example, the user's visual line-of-sight and thedefined location as the user moves the MID 600 (i.e., rotates the headup/down and/or right/left).

As the user looks away from the defined location, the OSP 632 respondsto the head motion signal by correspondingly moving the graphicalindicia across the display screen 608 maintaining visual alignment withthe defined location until it is no longer displayed when the definedlocation is no longer in the user's field of view, and similarly as theuser looks back toward the defined location the OSP 632 responds to thehead notion signal by making the graphical indicia reappear on acorresponding edge of the display screen 608 and then further trackmovement to maintain visual alignment with the defined location as theuser continues to rotate the HMD 600 toward the defined location. Inthis manner, a surgeon can virtually mark a location within a surgicalsite using a graphical indicia and can subsequently track that markedlocation based on the location of the graphical indicia within thedisplay screen 608 while the surgeon's head moves. The graphical indiciamay be included in a video stream from the camera 610 that iscommunicated, e.g., as a real-time video stream, to another HMD 600 wornby another person and/or that is communicated to the video server 650for recordation and/or forwarding to other devices.

The computer equipment 620 may compare patterns of objects in the videostream from the camera 610 viewing a patient's anatomy to patterns ofobjects in other video (e.g., images, etc.) from the video server 650and/or the imaging equipment 644 to identify levels of correspondence(e.g., output by a pattern matching algorithm). The computer equipment620 may display indicia in the display screen 608 responsive toidentifying a threshold level of correspondence between comparedobjects. For example, real-time video captured by the camera 610 duringsurgery of a patient may be processed by the computer equipment 620 andcompared to video captured by one or more other sources. The othersource(s) can include real-time feeds and/or earlier stored videoprovided by, for example, ultrasound equipment, cameras, CT scans, etc.The other source(s) may additionally or alternatively include ananatomical database specific for the particular patient or moregenerally for humans. The pattern matching may be constrained tocharacteristics of an object or a set of objects defined by a surgeon asbeing relevant to a present procedure. The computer equipment 620 maydisplay on the display screen 608 an indicia (e.g., a crosshair or colormarker) aligned with the identified object within the video from thecamera 610 to assist the surgeon with identifying a location within theobject of interest. The video sources may include an embedded markerthat indicates a location within an object that is of interest to thesurgeon. The pattern matching may further identify a location within thevideo stream from the camera 610 that corresponds to a location of themarker in the compared video, and an indicia may be displayed on thedisplay screen 608 aligned with the location within the video from thecamera 610 to assist the surgeon with identifying the location withinthe object of interest.

In some other embodiments. the user operates a handheld (e.g., wirelesscontroller) control panel and/or a foot control panel to providecommands or other input to control operation of the computer equipment620. A handheld control panel and/or foot control panel may be operatedto select among a plurality of video streams that are provided to theHMD 600 for viewing, control magnification of the video stream providedto the HMD 600, control the location of a graphical indicia displayedwithin a video stream, control stop-start recordation of a video streamfrom the camera 610, and/or control routing of video streams and otherinformation between the HMD 600, the video server 650, other HMDs 600,and other components of the system.

For example, a user may operate a portable computer, such as a laptopcomputer, tablet computer, mobile phone, or wearable computer to controlthe display of information on the display screen 608. For example, atablet computer may be configured to select among a plurality of virtualdisplay panels for display on the display screen 608. The tabletcomputer may select among the virtual display panels responsive to auser moving (e.g., rotating) the tablet computer, responsive toidentifying a hand gesture via a camera of the tablet computer and/orvia a touch sensitive display of the table computer, and/or responsiveto recognizing a voice command via a microphone of the tablet computer.A user may, for example, rotate the tablet computer horizontally toscroll through a plurality of virtual display panels that are virtuallyorganized along a horizontal plane. The user may similarly rotate thetable computer vertically to scroll through a plurality of virtualdisplay panels that are virtually organized along a vertical plane.

Virtual Displays Controlled by Head Mounted Display Apparatus

The computer equipment 620, via operation of the OSP 632, can generatevirtual display panels that display video received from the video server650 and/or other components of the system, and control which of thevirtual display panels are presently displayed on the display screen 608for viewing by a user based on user commands identified from motionsensed by the motion sensor 604, a gesture made by the user which issensed by the gesture sensor 602 and/or the camera 610, and/or a voicecommand via the microphone 612. The virtual display panels can bearranged and visually presented to the user to visually appear to floatwithin. space in front of the user. The user's head may be rotated upand down and/or right and left to observe content of different ones ofthe virtual display panels that appear to retain a static position inspace relative to the user's head. The user may alternatively oradditionally make a gesture, such as by moving a hand left or rightand/or moving the hand up or down, to cause the virtual display panelsto correspondingly slide left or right and/or up or down.

The display screen 608 may be controlled to display only one of thevirtual display panels at a time, such as by switching from one virtualdisplay panel to another adjacent virtual display panel, or display amore continuous panning across the virtual display panels through whicha user may view portions of two or more adjacent virtual display panels.

In case the user finds it undesirable to read the virtual display panelsmoving under control of the computer equipment 620 while looking around,the computer equipment 620 may enable the user to input a command (e.g.,“Lock”) which causes whichever virtual display panel is presently mostclosely spaced to the user's line-of-sight to be displayed full screenand held statically in-place not responding to head movement. Thevirtual display panel may remain statically locked as-displayed untilthe user deactivates the command via, for example, another command(e.g., “Unlock”).

The computer equipment 620 may be configured to provide anautomatic-lock and/or unlock of a virtual display panel relative to thedisplay screen 608. When an automatic mode is enabled, a virtual displaypanel becomes automatically locked relative to the display screen 608when the user's line-of-sight (e.g., yaw and pitch) indicated by thehead motion signal is within a first threshold amount (e.g., 2°) offsetfrom a defined location (e.g., yaw and pitch) of the virtual displaypanel. The computer equipment 620 may be configured to automaticallyunlock the virtual display panel from the display screen 608 when theusers line-of-sight (e.g., yaw and pitch) indicated by the head motionsignal becomes at least a second threshold amount (e.g., 5°), which isgreater than the first threshold amount, offset from the definedlocation (e.g., yaw and pitch) of the virtual display panel. Thethreshold amounts may be defined and/or adjusted by the user, and may bestored as a user's preference in the user's account information forsubsequent retrieval and use.

FIGS. 10-12 illustrate operations and methods that may be performed by asystem including a 750 to control the display of virtual display panelsthrough a display screen 752 of the HMD 750 according to someembodiments of the present disclosure. The various operations andmethods described in the context of FIGS. 10-12 are not limited to usewith the particular configuration of virtual display panels that areillustrated, but instead may be used with any number of virtual displaypanels and any arrangement of virtual display panels. Some of thevirtual display panels may be displayed immediately adjacent to oneanother to appear as a single larger virtual display panel. Moreover,although various embodiments are described in the context of aparticular HMD 750 configuration used in a surgical environment, theoperations and methods may be used with any HMD 750 configuration forany type of operational use.

The example embodiments of FIGS. 10-12 allow a surgeon or other user tosee several virtual displays of different medical information withoutlooking away from the surgical site and focusing far away to viewphysical monitors that may be mounted across the OR or elsewhereadjacent to the patient. In some embodiments, three operational “modes”of the virtual displays are selectively activated based upon pitch ofthe surgeon's head and the corresponding viewing line-of-sight of theuser. The three operations may be separately activated by increasingpitch angle of the HMD 750 through three corresponding ranges of viewingangles, such as low (directly at the surgical space), medium, high(horizontal eye-level). The viewing angle of the surgeon can bedetermined from the head motion signal output by a motion sensor of theHMD 750.

A full-screen operational mode is triggered when the OSP 632 determinesthat the surgeon is looking down at an operation site, which may bedetermined by when the pitch is below a first pitch threshold (e.g.,about −45°). The first pitch threshold may be defined and/or adjusted bythe surgeon based on a voice command, entered through a physical userinterface, etc. In the full-screen operational mode, a defined one ofthe video streams (e.g., a primary video stream received via HDMIchannel A) is displayed full screen through a display screen 752 of theHMD 750. The surgeon's corresponding preference settings may be saved ina configuration file stored in the memory 630 with an identifier for thesurgeon, so that the surgeon's preferred settings can be automaticallyretrieved upon recognition of the surgeon (e.g., via a login processthrough the computer equipment 620).

Referring to FIG. 10, the computer equipment 620, as illustrated in FIG.9, receives four video streams which it separately maps, via the OSP632, to four different virtual display panels 710, 720, 722, and 724.Alternatively, the computer equipment 620 may combine two or more videostreams to generate a combined video stream that is mapped to one of thevirtual display panels 710, 720, 722, and 724. The computer equipment620 may add graphical indicia and/or other information to one or more ofthe video streams, such as explained above. The computer equipment 620can arrange the virtual display panels in two-dimensional space or inthree-dimensional space. In the example of FIG. 10, three of the virtualdisplay panels 720, 722, and 724 are arranged horizontally along anupper row designated as secondary virtual display panels, and the fourthvirtual display panel 710 is arranged below the center virtual displaypanel 722 and designated as a primary virtual display panel. Otherarrangements of the virtual display panels 710, 720, 722, and 724 may beprovided, and other numbers of virtual display panels may be generatedby the computer equipment 620 with corresponding mapping to any numberof video streams.

In the example of FIG. 10, the surgeon's head is tilted downward belowthe first pitch threshold so that the surgeon's line-of-sight 702 istoward the operation site while looking through the TTL loupe. Thesurgeon may shift eye position upward so that the surgeon'sline-of-sight 700 now looks through the display screen 752. The computerequipment 620 responds to the surgeon's head tilted below the firstpitch threshold by operating in the full-screen operational mode toselect a defined one of the video streams which it displays on thedisplay screen 752 to generate the primary virtual display panel 710.Accordingly, the primary virtual display panel 710 can be positioned bythe HMD 750 to appear to the surgeon, while maintaining theline-of-sight 700, to hover in space above and adjacent to the locationof the operation site. None of the three other video streams mapped tothe three secondary virtual display panels 720, 722, and 724 arepresently displayed on the display screen 752.

FIG. 11 illustrates operations triggered by when the surgeon's headtilts up above the first pitch threshold and below a second pitchthreshold so the surgeon's line-of-sight is along line 704 through thedisplay screen 752. The primary virtual display panel 710 disappearsfrom the display screen 752 and the three video streams (e.g., HDMIchannels B, C and D) become separately viewable or at least partiallycollectively viewable as the three secondary virtual display panels 720,722, and 724 floating in space. These secondary virtual display panels720. 722, and 724 may first appear at default radial positions on asphere centered in the surgeon's head, but the surgeon can have theability to reposition and resize them for maximum convenience.

The collection of virtual display panels may be displayed as virtualfloating monitor devices that are stationary in location, so that theuser can move the HMD 750 to scan across their spaced apart locationsand see individual ones of the virtual display panels. The surgeon canscroll sideways across the secondary virtual display panels 720, 722,and 724 by making corresponding sideways (e.g., yaw) head movements, bymaking defined gestures, and/or by speaking defined voice commands. Thesurgeon can similarly scroll downward from the secondary virtual displaypanels 720, 722, and 724 to view the primary virtual display panel 710by making a corresponding downward (e.g., pitch) head movement, bymaking a defined gesture, and/or by speaking a defined voice command.

The computer equipment 620 may enlarge or shrink a portion of a videostream displayed on one of the virtual display panels being viewed by asurgeon responsive to a defined gesture by the surgeon and/or a definedvoice command by the surgeon.

The symbol generator 624 in combination with the OSP 632 processed bythe processor 626 of the computer equipment 620 may operate to display agraphical indicia (e.g., crosshair or other reticle) that can bepositioned within a presently viewed one of the virtual display panelsby the user responsive to recognition of voice commands via themicrophone 612, to track movement of the user's finger, hand, or otherobject operationally recognized by the gesture sensor 602 and/or in avideo stream from the camera 610. The user may trigger the OSP 632 tocapture a still image of the video stream displayed in the virtualdisplay plane with the incorporated graphical indicia responsive to avoice command, a gesture sensed by the gesture sensor 602, and/or amotion sensed by the motion sensor 604. In this manner, a surgeon mayview video within one of the virtual display panels and steer agraphical indicia to be aligned within the video overlapping apoint-of-interest, and trigger capture of a still image including thevideo and graphical indicia that can be saved on the video server 650and/or distributed to another HMD 600 for viewing by another surgeon orassistant.

For example, the user may trigger display a crosshair indicia responsiveto a voice command (e.g., “Crosshair On”). The graphical indicia mayinitially appear at the center of the display screen 608. Then, when theuser has repositioned the crosshair indicia on an item to be marked, theuser can speak another command (e.g., “Freeze”) to capture an image thatis shared through other HMDs and/or recorded in memory.

In one embodiment, the virtual display panels are maintained levelrelative to an artificial horizon and facing directly towards thesurgeon, and the surgeon can provide input to the computer equipment620, e.g., via the HMD 750, to separately adjust azimuth, elevation andsize parameters controlling display of each virtual display panel. Inanother embodiment, the surgeon can provide input to the computerequipment 620, e.g., via the HMD 750, to adjust position of the virtualdisplay panels in 6-degrees of freedom. The radius of the sphere(virtual distance of the virtual display panels from the display screen752 of the HMD 750) can be adjusted by the surgeon. A default focalradius of about 21 inches between the virtual display panels and thedisplay screen 752 of the HMD 750 may be used.

The size of individual ones of the virtual display panels and/or thesize of the collection of virtual display panels may be adjusted by thesurgeon. By default, each of the virtual display panels may be sized toapproximately fill the field-of-view of the display screen 752 of theHMD 750 (e.g., about 40° diagonal) when the surgeon is looking directlyat one of the virtual display panels. In this manner, one virtualdisplay panel can be controlled by the computer equipment 620 to fillthe field-of-view of the display screen 752 of the HMD 750 when thatdisplay is centered in the field-of-view of the surgeon. By default, thepositioning may be defined so that the secondary virtual display panel722 on the upper row is directly above the operation site (azimuth 0°and elevation about −10°). Another secondary virtual display panel 720starts just to the left of the secondary virtual display panel 722, andthe other secondary virtual display panel 724 starts just to the rightof the secondary virtual display panel 722. Distances between thevirtual display panels may be defined or adjusted by the surgeon, andthe surgeon's preferences may be stored associated with an identifierfor the surgeon.

FIG, 12 illustrates that when the surgeon's head tilts further upwardabove the second pitch threshold so that the surgeon looks along theline-of-sight line 706 above the zone where the virtual display panels710, 720, 722, and 724 are positioned, or anywhere else in the roomexcept towards the locations of the virtual display panels 710, 720,722, and 724, the display screen 752 does not display any of the fourvideo streams (e.g., blank display) such that the surgeon may, forexample, see that the display screen 752 without observing obstructingvideo. Thus, in FIG. 12 the virtual display panels 710 720, 722, and 724are illustrated as below the line-of-sight line 706 of the surgeon, andnone would be seen by the surgeon.

As explained above, the operations and methods disclosed herein are notrestricted to medical uses. These operations and methods are applicableto many other uses, including industrial uses such as warehousing,manufacturing, product inspection, and/or maintenance.

In one illustrative embodiment, a HMD configured for use in a warehouseenvironment can provide a plurality of virtual display panels containingdifferent information for guiding a user to a desired location withinthe warehouse and assisting the user with identifying a product on ashelf. For example, the HMD can respond to a user looking downward bydisplaying a first virtual display panel which provides directions fortraveling to the desired location within the warehouse. An example firstvirtual display panel may illustrated a virtual pathway (e.g., line) onthe ground that the user should follow to reach a desired location wherea product resides, etc. In response to the user looking upward, the HMDcan responsively display a second virtual display panel that assists theuser in identifying the product. An example second virtual display panelmay provide descriptive information and/or display a photograph orgraphical representation of the product.

Some or all of the operations and methods described herein as beingperformed by a and/or computer equipment may be performed by a portablecomputer, such as a laptop computer, tablet computer, mobile phone, dataterminal, or wearable computer (e.g., on wrist, arm, leg, etc.). Forexample, a tablet computer may be configured to select among a pluralityof virtual display panels for display on a display device of the tablecomputer and/or on a communicatively connected HMD. The tablet computermay select among the virtual display panels responsive to a user moving(e.g., rotating, pitching, etc.) the tablet computer, responsive toidentifying a hand gesture via a camera of the tablet computer and/orvia a touch sensitive display of the table computer, and/or responsiveto recognizing a voice command via a microphone of the tablet computer.A user may, for example, rotate the tablet computer horizontally toscroll through a plurality of virtual display panels that are virtuallyorganized along a horizontal plane. The user may similarly rotate thetable computer vertically to scroll through a plurality of virtualdisplay panels that are virtually organized along a vertical plane. Theportable computer (e.g., tablet computer) may be used to control displayof the virtual display panels on a communicatively connected HMD. Forexample, the user may make a hand gesture that is sensed by the portablecomputer (e.g., via touch sensitive display, proximity sensor, and/orcamera), the portable computer can communicate the command to computerequipment which then changes which of a plurality of virtual displaypanels are displayed on the HMD.

FIG. 13 is a block diagram of components of an augmented realitysurgical system that include a position tracking system 810 (e.g.,cameras spaced apart in the operating room) that track the location of asurgical tool 800 and/or prosthetic 802, the HMD 100, and a surgicalsite 804 or other target location on a patient. A computer system 820uses patient data from imaging equipment 830 to generate a twodimensional (2D) or three dimensional (3D) model. The imaging equipment830 may include, without limitation, x-ray equipment, endoscope cameras,magnetic resonance imaging equipment, computed tomography scanningequipment, three-dimensional ultrasound equipment, endoscopic equipment,and/or computer modeling equipment which can generate a multidimensional(e.g., 2D or 3D) model of a targeted site of a patient. The patient datacan include real-time feeds and/or earlier stored data from the imagingequipment 830, and may include an anatomical database specific for theparticular patient or more generally for humans.

The model can include reference markers or other references that assistwith performing correlations between virtual locations in the patientmodel and physical locations on the patient's body. The computer system820 uses the present locations of the HMID 100, the surgical site 804,and the surgical tool 800 and/or prosthetic 802 obtained by the positiontracking system 810 and uses the reference markers or other referencescontained in the patient model to transform the patient model to apresent perspective view of a wearer of the HMD 100. Some or all of thetransformed patient model can then be displayed on the HMD 100 toprovide the surgeon with a graphical overlay that is precisely orientedand scaled on the surgical site 804 or other target location on thepatient.

The computer system 820 may display graphical representations of thepatient model, the tool and/or the prosthetic on the display screen 110of the HMD 100, and animate movement of the displayed patient mode, tooland/or prosthetic to illustrate a planned procedure relative to adefined location of the surgical site 804 or other target location onthe patient's body. The HMD 100 may be communicatively connected to thecomputer system 820 through a wireless transceiver and/or wired networkinterface.

The computer system 820 may compare patterns of objects in the videostream from a camera on the HMD 100 to patterns of objects and/orreference markers in the patient model to identify levels ofcorrespondence, and may control transformation of the patient modelresponsive to identifying a threshold level of correspondence betweenthe compared objects. For example, real-time video captured by the HMDcamera during surgery of a patient may be processed by the computersystem 820 and compared to video captured by one or more other sources,e.g., the imaging equipment 830. The pattern matching may be constrainedto characteristics of an object or a set of objects defined by a surgeonas being relevant to a present procedure.

The computer system 820 can control transformation of the patient modelfor display on the display screen 110 based on the pattern matching. Thecomputer system 820 may display on the display screen 110 an indicia(e.g., a crosshair or color marker) aligned with an identified objectwithin the video from the HMD camera to assist the surgeon withidentifying the corresponding location on the patient. In oneembodiment, the computer system 820 displays a graphical indicia on thedisplay screen 110 aligned with one of the anatomical objects displayedon the display screen 110 from the rotated and scaled three dimensionalanatomical model responsive to identifying a threshold level ofcorrespondence between a pattern of the one of the anatomical objectsand a pattern of one of the anatomical objects in the video stream fromthe video camera.

The computer system 820 may similarly receive other data and videostreams from a patient database and other electronic equipment, whichcan be selectively displayed on the MID 100, As used herein, a videostream can include any type of information that can be provided to adisplay device for display, including without limitation a still image(e.g., digital photo), a sequence of still images, and video havingframes provided at a defined frame rate. The computer system 820 canretrieve information relating to a patient's medical history and data.obtained by real-time monitoring of a patient, including, for example,hemodynamic, respiratory, and electrophysiological signals.

Although the computer system 820 is illustrated as being separate fromthe HMD 100, some or all of the operations disclosed herein as beingperformed by the computer system 820 may additionally or alternativelybe performed by one or more processors residing within the HMD 100.Similarly, some of the operations disclosed herein as being performed bythe HMD 100 may additionally or alternatively be performed by one ormore processors residing within the computer system 820.

Although various embodiments are disclosed in the context of a WAD someother embodiments are directed to tracking the location of a handhelddisplay, such as a tablet computer, and displaying the transformedpatient model on the handheld display. The handheld display may includetracking markers, such as on a front surface and/or back surface of thedisplay housing, which are tracked by the position tracking system 810.The handheld display may include a camera that provides a video signalthat is combined with a video signal of the transformed patient modelfrom the computer system 820, to display a real world view of thepatient augmented with the graphical overlay from a portion of thetransformed patient model.

The transformed patient model may additionally be relayed to other HMDs100 worn by other personnel assisting with the procedure, to otherdisplay devices, and/or to a video server for storage. In this manner,other personnel may observe what the surgeon views on the display screen110.

FIG. 14 is another block diagram of the electronic components of theaugmented reality surgical system of FIG. 13 according to someembodiments of the present disclosure. Referring to FIG. 14, theposition tracking system 810 can include a camera system 902 that tracksthe location tracking markers 904 attached to a surgery table, trackingmarkers 906 attached to patient adjacent to a surgical or other targetsite, tracking markers 908 attached to a surgery tool and/or prosthetic,and tracking markers 910 attached to the HMD 100. The camera system 902can include a plurality of cameras that are spaced apart at definedlocations within an operating room and each having a field of view thatcan observe objects to be tracked. In the illustrated example, thecamera system 902 includes two sets of cameras spaced apart by a knowndistance and relative orientation. The position tracking system 810 mayuse active optical markers (e.g., light emitting sources), passiveoptical markers (e.g., light reflectors), but is not limited to the useof cameras. The position tracking system 810 may additionally oralternatively use electromagnetic ranging and trackers, ultrasonicemitters/sensors for ranging and trackers, etc.

Positioning data of the HMD 100 can include navigation coordinate systemdata. determined from the location of the tracking markers 910 attachedto the HMD 100, and inertial coordinate system data from the inertialsensors. The navigation coordinate system data and the inertialcoordinate system data can be compensated for initial calibration anddrift correction over time by a calibration component 912 and combinedby a fusion component 914 to output combined HMD position data.

A relative positioning component 916 identifies the relative positionand angular orientation of each of the tracked markers 904-910 and thecombined HMD position data. The component 916 may perform coordinatetransformations of the relative coordinate systems of the surgery table,the patient, the surgery tool, and the HMD 100 to a unified (common)coordinate system. In one embodiment, the relative positioning component916 outputs sight coordinates data, patient model coordinates data, andtool coordinates data to an image generator 918. The sight coordinatesdata can be generated based on the combined HMD position datatransformed to the unified coordinate system. The patient modelcoordinates data can be generated based on the position and orientationof the tracking markers 906 attached to the patient transformed to theunified coordinate system, and the tool coordinates data can begenerated based on the position and orientation of the tracking markers908 attached to the surgery tool transformed to the unified coordinatesystem.

The image generator 918 transforms the patient model (e.g., from theimaging equipment 830) to a present perspective view of a wearer of theHMD 100 mapped to a corresponding object within the FOV of the displayscreen 110 which is modeled by the patient model. The image generator918 provides video generated based on the transformed patient model tothe HMD 100 for display as a visual model that is dynamically orientedand scaled as a graphical overlay on the surgical site 804 or elsewhereto a corresponding location on the patient where the wearer of the HMD100 is presently looking and which contains a corresponding object whichis modeled by the patient model.

For example, the image generator 918 determines whether any portion ofthe patient's body is presently within the field of view of what thesurgeon sees through the display screen 110 that corresponds to anyportion of the transformed patient model. When a portion of thetransformed patient model corresponds to a portion of the patient's bodywithin the surgeon's field of view through the display screen 110, theimage generator 918 generates video for display on the display screen110 based on the corresponding portion of the transformed patient model,while translating the portion of the transformed patient model andscaling size of the portion of the transformed patient model to providean accurately scaled graphical representation of the object that wasimaged from the patient or modeled from another source such as ananatomical database.

Thus, for example, when a surgeon's head is rotated so that a portion ofa patient's body having a bone that is modeled through CT imagery databecomes within the field of view of the display screen 110, the imagegenerator 918 transforms and scales the patient model of the bone togenerate a graphical representation of the bone that is displayed in thedisplay screen 110 as a graphical overlay that matches the orientationand size of the bone from the perspective of the surgeon as-if thesurgeon could view the bone through intervening layers of tissue and/ororgans.

In the example illustration of block 920, a leg bone model that has beengenerated, e.g., based on a CT scan of the leg, is transformed anddisplayed on the display screen 110 to have the accurate six degree offreedom orientation and size relative to the leg bone when viewed as agraphically illustrated representation 922 of the leg bone superimposedon a skin surface of the leg. The surgeon therefore sees the skinsurface of the leg through the semitransparent display screen 110 of theHMD 100 with an graphically illustrated representation of the leg bonemodel overlaid thereon.

Although the graphical representation 922 of the leg bone is illustratedas being displayed in a superimposed position on a skin surface of theleg, the graphical representation 922 can be displayed at otherlocations which may be controllable by the surgeon. The surgeon may, forexample, select to have the graphical representation 922 displayed witha defined offset distance above or below the leg. Moreover, the surgeonmay control the size of the displayed graphical representation 922relative to the leg. The surgeon may, for example, temporarily magnifythe displayed graphical representation 922 to view certain details andthen return the displayed graphical representation 922 to be scaled andaligned with the leg.

FIG. 15 is another block diagram that illustrates further exampleoperations of the electronic components of the augmented realitysurgical system of FIG. 13 according to some alternative or additionalembodiments of the present disclosure. Referring to FIG. 15, the ImageryDevice (IMG) 1000 provides x-ray imagery data, e.g. 2D or 3D pictures,with or without reference markers for aiding with location correlation.The image generator 1000 may correspond to the imaging equipment 830 ofFIG. 13. The imagery data can be generated before and/or during anoperation. For example, real-time generated imagery data from an x-raydevice may be used to aid a surgeon with navigating a tool to a targetsite within a patient during on operation. In another example, 3Dimagery data generated days or weeks before an operation can be reusedand combined with 2D or 3D x-ray imagery data generated real-time duringthe operation. The imagery data can be output as a voxelised object,with an accurate 3D geometry of the imaged object.

Calibration of the x-ray device may be performed using an instrumentedreference cube. Using an automatic superposition process, all thetransformation tables can be determined based on the calibration toprovide a clean and rectified geometry of the scanned object, withoutundesirable deformation. The calibration tables are specific to eachx-ray device.

The REF component 1002 accurately references (correlates) the voxelisedobject in the virtual world to the tracked objects in the real worldwhich may be tracked using the tracking markers explained above or otherapproaches. The x-ray imagery data can include reference markers whichcan be created using small objects placed on the imaged object (e.g.,patient's body) and which can be readily identified in the x-rayimagery. The reference markers may be invasively attached to the patient(e.g., implanted during an operation by a surgeon) and/or may benon-invasively attached (e.g. adhesively attached to skin).

A modeling component (MOD) 1004 transforms the x-ray imagery data to a3D patient model (graphical model) which can include reference markers.The patient model may have several separate representations, such as: 1)bones only; 2) skin only; 3) bones and skim 4) wireframe or color; etc.In case of available articulation between two bones, geometricaltransformation between the bones may be defined in the patient model toenable animated movement thereof. The reference markers on articulatedbones or other attached objects are referenced in the same coordinatesystem as the body and/or skeletal model. The number of referencemarkers may be at least three per rigid object, e.g., bone.

A navigation component (NAV) 1006, which may be a component of theposition tracking system 810 of FIG. 14, generates the relativepositions (e.g., X, Y, Z coordinate system) of: 1) the targeted bodypart(s), 2) the HMD 100; 3) surgical tools and/or auxiliary tools; etc.The NAV 1006 identifies the relative positions of the tracked markers,and may further determine the relative angular orientations of thetracked markers 904, 906, 908, and 910. An inertial measurement unit(IMU) 1008 can be mounted to the HMD 100 and configured to sensetranslational and/or rotational movement and/or static orientation ofthe surgeon's head, and the sensor data is converted to position dataand angular orientation data (e.g., in the inertial coordinate system)by a head tracking (HTK) component 1009. The head tracking component1009 may perform Kalman filtering on the sensor data to filter noiseand/or sensor drift over time when generating the position data andangular orientation data.

A generator component (GEN) 1010 computes in real-time the transformedpatient model to be displayed (3D models, symbology, etc.) according torelative positions of the various tracked objects in the operating room.The graphical representation provided by the transformed patient modelcan be monoscopic or stereoscopic, and can represent several differentmodes (e.g., wireframe, color, textured, etc.) that are selectable undersurgeon control. The GEN 1010 may perform calibration operations for theNAV 4\1006, the display screen (DIS) 110, the head tracking component1009, and man-machine interface (e.g., calibration of gesture basedcontrol operations, voice based control operations, etc.).

As explained above, the display screen (DIS) 110 can be a see-throughdisplay device that is fed video which displays in real-time portions ofthe transformed patient model and desired symbology, which can beaccurately superimposed on the target surgical site or other target areaof the patient within the FOV of the surgeon.

FIG. 16 is a block diagram that illustrates further example operationsof an electronic system subcomponent that can be included in theaugmented reality surgical system of FIG. 13 according to some otherembodiments of the present disclosure.

Referring to FIG. 16, the subcomponent accesses CT-SCAN data stored in aDICOM file 1100. The example file 1100 contains raw 3D CT scan datawhich is cleaned, filtered, and processed through the operations ofblocks 1102-1118 to generate a 3D model that is ready for graphicalvisualization and display on the HMD 100. The geometric structure of the3D CT scan is based on a set of parallel slices along planes through thepatient's body. The slices can have a constant thickness and a sharedcoordinate system.

Although various embodiments are described in the context of processing3D CT scan data to generate a graphical model for illustration to anoperator, these embodiments are not limited thereto and may be used withany type of patient data that can be graphically modeled forillustration.

The system can adapt its processing of the 3D CT scan data based onknowing what is characterized by the data (e.g., one or more definedorgans, one or more defined skeletal structures). The type ofinformation that is extracted from the data can be selected by theoperator based on what is desired for graphical visualization throughthe HMD 100 and/or the type of information may be automaticallyselected. One selection criterion is based on the X-ray intensity ofeach pixel per slice. Anatomical structure can be identified andcharacterized using windowing operations (block 1102) that map X-rayintensity to a contrast intensity image based on parameters that arehave predefined for identifying skeletal structure, muscular structure,organ structure, vascular structure, etc. In the embodiment of FIG. 16,the windowing operations processes the types of CT scan data thatincludes: CT-ABDOMEN, CT-CRANE CT-LUNG, CT-PELVIS, CT-BONE, etc. Outputof the windowing operations can be voxelised (block 1104), e.g., by datavectorization based on opacity of the material.

Image processing (block 1106) can then be performed that includescontrast adjustment, Gaussian noise filtering, etc. Selection are made(block 1108) among various available types of modeled iso-surfaces, suchas bone, outer skin, organs, etc., based on operator (e.g., surgeon)input and/or other selection criteria. The various types of iso-surfacemodels may be generated. using data from the voxel model 1110. A 3Dmodel is generated (block 1112) based on the selections. Calibrationinformation can be generated (block 1114) to facilitate the set-up ofthe system by performing an initial calibration of the orientation ofmarkers located on the HMD 100, while the operator wears the HMD 100,markers located on the patient, markers located on tools, etc., such asdescribed above regarding FIG. 14. The CT-scan can be performed Whilemarkers are fixed to the patient, such as described above. Locations ofthe markers are correlated to the single coordinate system for all thederived models (voxel or mesh).

Smoothing operations are performed (block 1116) on the resulting data,and data validity checks and operator selective controls are applied(block 1118). A 3D mesh graphical model is generated and can bedisplayed (block 1120) through the HMD 100 to illustrate the relativeposition and orientation of a surgical tool, the modeled iso-surfacesfrom the 3D CT scan data, and any other operator selected and/or definedinformation. The same coordinate system can be maintained throughout theoperations of blocks 1102-1120.

As explained above, the system can generate a graphical model from thepatient data that represents detectable anatomical structure that ishidden from direct observation. The system can automatically orient,scale, and display the model so that it is viewable through the HMD 100superimposed on the relevant area of the physical anatomy of thepatient. Thus, for example, when a surgeon's head is rotated so that anarea of a patient's body having a bone modeled through CT imagery databecomes within We field of view of the display screen 110 of the HMD100, the system displays a graphical model of the bone and selectedtypes of anatomical structure (e.g., skeletal structure, muscularstructure, organ structure, vascular structure, etc.). The surgeon isthereby able to peer into the patient to view a representation of thebone and/or intervening layers of tissue, organs, etc.

As will be appreciated in view of the present disclosure, previouslyavailable procedures that required a surgeon to view static CT scans orother patient data on remote monitors provided limited guidance to thesurgeon for where a drill or other surgical tool should be placed on thepatient's body and, more particularly, how the drill should be orientedrelative to the patient's body so that the drill bit will travel throughacceptable structure in the patient's body and reach a target location.Some further embodiments of the present disclosure are directed todisplaying additional information through the HMD 100 that providesreal-time feedback to the surgeon for where the drill or other toolshould be located and oriented before continuing a procedure.

The system can compute and graphically display in real-time through theHMD 100 a projected path that the drill would make through a patient'sbody based on a present orientation and location of the drill. Thesurgeon can be allowed to select from among several availablevisualization modes to have particular anatomical structure and thetrajectory of one or more surgical tools graphically modeled anddisplayed to the surgeon through the HMD 100 based on their computedrelative locations, orientations and scaled relationships.

FIG. 17 is a block diagram that illustrates further example operationsof an electronic system subcomponent 1200 that can be included in theaugmented reality surgical system of FIG. 13 to generate the graphicalrepresentations of FIGS. 18-23 in according with some embodiments of thepresent disclosure. FIG. 18 illustrates a graphical image generated onthe HMD 100 that shows a virtual trajectory 1312 of a drill bit 1310extending from a drill or other surgical tool (e.g., scalpel, etc.) intoa patient's anatomical bone model 1302 and other selected interveningstructure, and which is overlaid at a patient site 1300.

Referring to FIGS. 17 and 18, a tracking system 1208 tracks the relativelocation and orientation of reference markers attached to the drill bit1310, the patient site 1300, and the HMD 100, as illustrated in FIG. 14,and computes the relative distance and orientation therebetween. Thetracking system 1208 may operate based on the position tracking system810 described above in FIG. 14, A slice builder component 1204 uses therelative distance and orientation between the drill bit 1310, thepatient site 1300, and/or the HAM 100 to select data from the voxelmodel 1110 that is used to form a slice for display through the HMD 100.Data in the slice may directly correspond to data contained in the voxelmodel 1110 and/or may be generated from voxel model 1110 data which wasobtained from a plurality of 3D CT scans which are intersected by thevirtual trajectory 1312 of the drill bit 1310. A viewer component 1206combines the slice with the 3D mesh model 1202 and displays the combinedgraphical rendering on the HMD 100 so that it is precisely superimposedon the patient site 1300 from the surgeon's point-of-view. The viewercomponent 1206 also graphically displays the virtual trajectory 1312,which may be computed using operations based on a ray-tracing typealgorithm. In the example of FIG. 18, the viewer component 1206 has alsodisplayed on the HMD 100 a cross sectional slice 1320 that passesthrough the target location 1314.

The virtual trajectory 1312 can be recomputed and dynamically displayedat a sufficient update rate to provide real-time feedback to a surgeonwho is repositioning and reorienting the drill bit 1310 and/or thepatient site 1300 so that the virtual trajectory 1312 will intersect atarget location 1314. In the example of FIG, 18, the target location1314 corresponds to a point where the drill bit 1310 would impact thegraphically displayed bone model 1302, which is oriented and scaled tomatch the patient's bone. A sufficient update rate for recomputing anddisplaying the virtual trajectory 1312 to provide acceptable real-timefeedback to a surgeon may be, without limitation, at least 5 Hz.

The system subcomponent 1200 can be configured to provide definedvisualization modes that a surgeon can select among using head movement,hand gestures, voice commands, etc., which are sensed by the HMD 100. Insome visualization modes the surgeon controls what type of anatomicalstructure is graphically displayed on the HMD 100. For example, thesurgeon can select among various visualization modes that control whichone or more of the following are displayed on the HMD 100: 1) bone; 2)skin; 3) muscle; 4) organ; 5) vessel; 6) virtual tool trajectory; and 7)cross sectional slice. Another visualization. mode can control whichcross sectional slices are displayed and/or the orientation of theslice(s) relative to the surgeon's point-of-view. Some visualizationmodes cause the system subcomponent 1200 to graphically renderanatomical structure of the patient for display as wireframes, polygons,or smoothed surfaces, and which can be selectively displayed inmonochrome or false colors. A surgeon can dynamically command switchingbetween the available visualization modes and can cause any combinationof two more of the visualization modes to be simultaneously active.

In the example of FIG. 18, the surgeon has selected a visualization modethat displays the slice located at the target location 1314. The slicemay be automatically generated and displayed by the viewer 1206responsive to the target location 1314 being computed based on where thedrill bit 1310 will intersect the surface of the graphically displayedbone model 1302 or other selected anatomical structure. FIG. 19illustrates a portion of the slice 1320 along plane 19-19 in FIG. 18that has been rotated to provide a front view. The surgeon may selectanother visualization mode that controls the system subcomponent 1200,e.g., through rotation of the surgeon's head, to rotate the slice 1320displayed on the HMD 100 from the orientation shown in FIG. 18 to theorientation shown in FIG. 19 along the illustrated x-y coordinate plane,and/or to any other orientation. The slice 1320 can be variably scaled(e.g., zoom-in or zoom-out) for display responsive to commands receivedfrom the surgeon. The surgeon can select among various views disclosedhere to guide the surgeon's manipulation of the drill hit 1310 or othersurgical tool.

The surgeon can select another visualization mode that causes the systemsubcomponent 1200 to simultaneously or sequentially display a sequenceof cross-sectional slices that are spaced apart along the virtualtrajectory 1312 of the drill bit 1310. An example graphical displaygenerated on the HMD 100 according to this visualization mode is shownin FIG. 20. In the non-limiting example of FIG. 20, three slices 1322,1324, 1326 are parallel to one another, e.g., parallel to theillustrated x-y coordinate plane, and are spaced apart and each centeredalong the virtual trajectory 1312. The graphical display shows thespatial relationship between the virtual trajectory 1312 and anatomicalstructure illustrates in the slices 1322, 1324, 1326, and moreover showsintersection points between the virtual trajectory 1312 and each of theslices 1322, 1324, 1326. The surgeon can control the system subcomponent1200 to rotate and/or scale all three slices 1322, 1324, 1326 or aselected one or more thereof. The slices are selected based on thecomputer determining that the slice contains imaging data of a locationwithin the patient that is intersected by the virtual trajectory 1312.The surgeon can control how many slices are simultaneously displayed andcan control where a slice is generated and displayed along the virtualtrajectory 1312. For example, through a head movement, hand gesture, orother command the surgeon may move a pointer along the virtualtrajectory 1312 to select a location where a slice, e.g., along the x-yplane, is to be generated and displayed on the HMD 100.

The surgeon can select another visualization mode that causes the systemsubcomponent 1200 to simultaneously or sequentially display anothersequence of cross-sectional slices that are spaced apart along thevirtual trajectory 1312 of the drill bit 1310 and oriented in parallelplanes that are perpendicular to the virtual trajectory 1312. An examplegraphical display generated on the HMD 100 according to thisvisualization mode is shown in FIG. 21. In the non-limiting example ofFIG. 21, three slices 1330, 1332, 1334 are parallel to one another andspaced apart and each centered along the virtual trajectory 1312. Thesurgeon can control the system subcomponent 1200 to rotate and/or scaleall three slices 1330, 1332, 1334 or a selected one or more thereof. Thesurgeon can control how many slices are simultaneously displayed and cancontrol where a slice is generated and displayed along the virtualtrajectory 1312, and may control how many dimensions are illustrated inthe various view and/or can select among various perspective views. Thesurgeon may furthermore control whether the graphical display renders athree dimensional or orthographic projection view of the graphicalanatomical model and/or slices thereof.

The visualization mode that is displayed in FIG. 21 provides the surgeonwith a view from the perspective of looking-down the drill bit 1310. Agraphical display illustrates to the surgeon the virtual trajectory 1312and a tool impact location 1314 where the drill bit 1310 is projected tomake contact with the bone if the surgeon proceeded to drill whilemaintaining the present orientation of the drill bit 1310. The graphicaldisplay may also illustrate a target location 1316 that the surgeon hasearlier defined as being a desired location where the drill bit 1310should contact the bone. Simultaneous illustration of the tool impactlocation 1314 and the target location 1316 in this manner can be aparticularly useful view mode while a surgeon is seeking to position andorient the drill bit 1310 to intersect or to avoid intersecting variousanatomical structure illustrated by one or more of the slices 1330,1332, 1334 and/or other slices selected by the surgeon, and whileaccomplishing the objective of guiding the drill bit 1310 to contact thetarget location 1316 on the bone. Through a head movement, hand gesture,or other command the surgeon may move a pointer along the virtualtrajectory 1312 to select a location where a perpendicular slice is tobe generated and displayed on the HMD 100.

In a further visualization mode, the surgeon can control the systemsubcomponent 1200 to rotate and/or scale one or more of the slices 1330,1332, 1334. FIG. 22 illustrates the slice 1334 along plane 22-22 in FIG.21 rotated to provide a front view, and illustrates the tool impactlocation 1314 where the drill bit 1310 would impact the graphicallydisplayed hone model 1302 based on a present location and orientation ofthe drill bit 1310 relative to the patient site 1300 and furtherillustrates the target location 1316. FIG. 23 illustrates the slice 1330along plane 23-23 in FIG. 21 rotated to provide a front view, andillustrates the tool impact location 1314 where the drill bit 1310 wouldimpact the graphically displayed hone model 1302 based on a presentlocation and orientation of the drill bit 1310 relative to the patientsite 1300 and further illustrates the target location 1316. The surgeonmay control the system subcomponent 1200, e.g., through rotation of thesurgeon's head, to rotate and/or scale one or more of the slicesdisplayed on the HMD 100. The surgeon may furthermore control the systemsubcomponent 1200 to selectively display any one or more of the virtualtrajectory 1312, the tool impact location 1314, and the target location1314.

In this manner various embodiments of the present disclosure displayinginformation through the HMD 100 that provides real-time guidance andfeedback to a surgeon who seeks to position and orient a surgical toolto reach a target location within a patient.

Further Definitions and Embodiments

In the above-description of various embodiments of the presentdisclosure, aspects of the present disclosure may be illustrated anddescribed herein in any of a number of patentable classes or contextsincluding any new and useful process, machine. manufacture, orcomposition of matter, or any new and useful improvement thereof.Accordingly, aspects of the present disclosure may be implemented inentirely hardware, entirely software (including firmware, residentsoftware, micro-code, etc.) or combining software and hardwareimplementation. that may all generally be referred to herein as a“circuit,” “module,” “component,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productcomprising one or more computer readable media having computer readableprogram code embodied thereon.

Any combination of one or more computer readable media may be used. Thecomputer readable media may be a computer readable signal medium or acomputer readable storage medium. A computer readable storage medium maybe, for example, but not limited to, an electronic magnetic optical,electromagnetic, or semiconductor system, apparatus, or device or anysuitable combination of the foregoing. More specific examples (anon-exhaustive list) of the computer readable storage medium wouldinclude the following: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an appropriateoptical fiber with a repeater, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable signal medium may be transmitted usingany appropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB NET,Python or the like, conventional procedural programming languages, suchas the “C” programming language, Visual Basic, Fortran 2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby andGroovy, or other programming languages. The program code may executeentirely on on the user's computer and partly on a remote computer orentirely on the remote computer or server. In the latter scenario, theremote computer may be connected to the user's computer through any typeof network, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider) or ina cloud computing environment or offered as a service such as a Softwareas a Service (SaaS).

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a. computerreadable medium that when executed can direct a computer, otherprogrammable data processing apparatus, or other devices to function ina particular manner, such that the instructions when stored in thecomputer readable medium produce an article of manufacture includinginstructions which when executed, cause a computer to implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable instruction execution apparatus, or otherdevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatuses or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

It is to be understood that the terminology used herein is for thepurpose of describing particular embodiments only and is not intended tobe limiting of the invention. Unless otherwise defined, all terms(including technical and scientific terms) used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this disclosure belongs. It will be further understood that terms,such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of this specification and the relevant art and will not beinterpreted in an idealized or overly formal sense unless expressly sodefined herein.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousaspects of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularaspects only and is not intended to be limiting of the disclosure. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. it will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items. Like reference numbers signify like elements throughoutthe description of the figures.

The corresponding structures, materials, acts, and equivalents of anymeans or step plus function elements in the claims below are intended toinclude any disclosed structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present disclosure has been presentedfor purposes of illustration and description. but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of thedisclosure.

The aspects of the disclosure herein were chosen and described in orderto best explain the principles of the disclosure and the practicalapplication, and to enable others of ordinary skill in the art tounderstand the disclosure with various modifications as are suited tothe particular use contemplated.

What is claimed is:
 1. An augmented reality surgical system comprising:a display comprising a see-through display screen that display imageswhile allowing transmission of ambient light therethrough; a motionsensor connected to the display and configured to output a motion signalindicating measured movement of the display; and at least one cameracoupled to a portion of the display and configured to observe referencemarkers connected to a patient, and reference markers connected to asurgical tool located within a surgical room; computer equipmentconfigured to: compute the relative location and orientation of thedisplay and the reference markers connected to the patient based onprocessing an image signal from the at least one camera; generate athree dimensional anatomical image using patient data created by medicalimaging equipment that has imaged a portion of the patient, rotate andscale at least a portion of the three dimensional anatomical model basedon the relative location and orientation of the reference markersconnected to the head mounted display and the reference markersconnected to the patient; and generate a video signal based on at leasta portion of the three dimensional anatomical image and the location andorientation of the reference makers coupled to the patient and asurgical instrument, and output a video signal to the display screen onthe display, wherein the video signal is a graphical representation of avirtual trajectory of the surgical instrument that intersects a targetlocation on a patient, and wherein the virtual trajectory is acontinuously updated and dynamically displayed to a surgeon who isrepositioning and reorienting the surgical instrument.
 2. The augmentedreality surgical system of claim 1, wherein the user sees a graphicalrepresentation on the display screen of the at least a portion of thethree dimensional anatomical model oriented and scaled to provide adisplayed above the patient that was imaged by the medical imagingequipment.
 3. The augmented reality surgical system of claim 1, whereinthe computer equipment compares patterns of anatomical objects in avideo stream from at least one video camera coupled to the surgicalsystem to patterns of anatomical objects in the patient data created bythe medical imaging equipment, and controls generation of the threedimensional anatomical model from the patient data responsive toidentifying a threshold level of correspondence between the comparedpatterns of anatomical objects.
 4. The augmented reality surgical systemof claim 1, wherein the computer equipment compares patterns ofanatomical objects in a video stream from at least one video camera topatterns of anatomical objects in the patient data created by themedical imaging equipment, and displays a graphical indicia on thedisplay screen aligned with one of the anatomical objects displayed onthe display screen from the rotated and scaled three dimensionalanatomical model responsive to identifying a threshold level ofcorrespondence between a pattern of the one of the anatomical objectsand a pattern of one of the anatomical objects in the video stream fromthe at least one video camera.
 5. The augmented reality surgical systemof claim 1, wherein: the at least one camera is further configured toobserve reference markers connected to a surgical tool located within asurgical room; and generate the video signal to include a graphicalrepresentation of the surgical tool illustrated at a position relativeto the three dimensional anatomical model that is determined based onthe relative location and orientation of the reference markers connectedto the head mounted display, the reference markers connected to thepatient, and the reference markers connected to the surgical tool. 6.The augmented reality surgical system of claim 5, wherein the computerequipment is further configured to: generate a graphical representationof a virtual trajectory extending from the surgical tool into the threedimensional anatomical model based on the relative location andorientation of the reference markers connected to the head mounteddisplay, the reference markers connected to the patient, and thereference markers connected to the surgical tool; and generate the videosignal to include the graphical representation of the virtualtrajectory.
 7. The augmented reality surgical system of claim 6,wherein: the computer equipment is further configured to select an imageslice from among a plurality of image slices contained in the threedimensional anatomical model, the image slice being selected based onthe computer equipment determining that the image slice is traversed bythe virtual trajectory extending from the surgical tool, and to generatethe video signal to include the image slice.
 8. The augmented realitysurgical system of claim 7, wherein a graphical representation of theimage slice and the graphical representation of the virtual trajectorythat are displayed on the display screen are responsive to the headmotion signal from the motion sensor and/or responsive to a commandreceived from a user.
 9. The augmented reality surgical system of claim7, wherein a graphical representation of the image slice and thegraphical representation of the virtual trajectory that are displayed onthe display screen to provide a view of the image slice that isperpendicular to a direction of the virtual trajectory.
 10. Theaugmented reality surgical system of claim 9, wherein: the computerequipment is further configured to identify a target location within theimage slice, and to display the target location relative to thegraphical representation of the virtual trajectory.
 11. An augmentedreality surgical system for displaying multiple video streams to a user,the augmented reality surgical system comprising: a display comprising asee-through display screen that display images while allowingtransmission of ambient light therethrough a motion sensor connected tothe display and configured to output a motion signal indicating measuredmovement of the display; and a position tracking system configured totrack the location of the surgical tool, and/or a prosthetic, thedisplay, a surgical site and a target location of the patient, whereinthe video signal is a graphical representation of a virtual trajectoryof the surgical instrument that intersects a target location on apatient, and wherein the virtual trajectory is a continuously updatedand dynamically displayed to a surgeon who is repositioning andreorienting the surgical instrument.
 12. The augmented reality surgicalsystem of claim 11, wherein: the computer equipment is communicativelyconnected to a surgical video server to receive the video streams, oneof the video streams comprises data generated by medical imagingequipment, another one of the video streams comprises information from apatient database defining a patient's medical history, and another oneof the video streams comprises data generated by medical equipment basedon real-time monitoring of the patient's vitals.
 13. The augmentedreality surgical system of claim 11, wherein: the computer equipment isconfigured to select one of the video streams from among the videostreams based on the motion signal, and to output the selected one ofthe video streams as a video signal to the display screen.
 14. Theaugmented reality surgical system of claim 11, wherein the motion sensoris configured to output the motion signal containing a pitch componentthat provides an indication of pitch angle of the head mounted display.15. The augmented reality surgical system of claim 11, wherein thecomputer equipment recognizes voice commands and/or gesture commandsfrom the user, and defines and/or adjusts coordinates of virtualfloating panels displayed through the display screen responsive to therecognized voice commands and/or gesture commands.
 16. The augmentedreality surgical system of claim 11, further comprising a gesture sensorelectrically coupled to the display and configured to provide a gesturesignal to the computer equipment, wherein the computer equipment isconfigured to identify a gesture made by the user responsive to thegesture signal and perform a command associated with the identifiedgesture, the command controls which of the video streams are output as avideo signal to the display screen.
 17. The augmented reality surgicalsystem of claim 16, wherein the gesture sensor comprises a video camerathat outputs the gesture signal to the computer equipment.
 18. Theaugmented reality surgical system of claim 16, wherein the gesturesensor comprises a photoelectric motion sensor and/or a proximity sensorcomprising at least infrared emitter that is adjacent to at least onephotodiode, the at least one photodiode senses infrared light emitted bythe infrared emitter which is reflected back by an object placed by theuser within a field of view of the at least one photodiode, the at leastone photodiode outputting the gesture signal to the computer equipment.19. The augmented reality surgical system of claim 16, Wherein thegesture sensor comprises one or more ultrasonic echo ranging transducersthat output the gesture signal to the computer equipment responsive tobouncing an ultrasonic signal off an object placed by the user withinrange of the one or more ultrasonic echo ranging transducers.
 20. Theaugmented reality surgical system of claim 11, wherein the computerequipment communicates video from the video camera through a network toanother display worn by another person.